fix: export visible replay completions - #869

Issue

Fixes #854.

Description of changes

A callback can complete while an execution is suspended, then be skipped when replay returns its parent's cached result. Both OTel views now retain visible terminal updates from invocation input and checkpoint changes, deliver them through the existing terminal span path, and export unvisited completions before Invocation closes.

Parent registration drains deferred children under the actual replay segment. Completed parent contexts remain available for later updates, and finalization materializes pending ancestors before descendants. Successful observations are deduplicated within an invocation; each newly opened Invocation-view ReplayChildren or virtual-context segment still receives its own natural End.

Design and simplification

• A private CompletionState owns pending updates, the parent index, in-flight claims, successful-delivery bookkeeping, and cleanup for both views. Completion payloads have one authoritative dictionary; the parent index stores only IDs and preserves observation order. Completed input parents are retained as an ID set.
• Each plugin keeps its own parent selection, span rendering, and replay-segment policy. State access and parent-registration dequeue use the existing span-registry lock. Tracer and processor callbacks run outside it, and delivery is recorded only after span completion succeeds.
• Finalization uses standard-library topological ordering. Completed parent references preserve identity but do not count as exported terminal spans. Child processor failures during parent registration are logged without interrupting attachment of the parent's context; undelivered metadata remains pending.
• Simpler alternatives considered: an invocation-end-only scan cannot attach children while their actual replay parent is active; treating the full snapshot as new work repeats ordinary replay exports; synthesizing core End hooks changes the contract for every plugin. Sharing the delivery state removes duplicated transitions without combining the two views' different span lifecycles.

The parent index tracks only children awaiting delivery. Completing one child removes only its ID; completing the last indexed child removes the parent bucket. Keeping empty buckets until invocation teardown would avoid the pruning branch but retain parent-index entries with no pending child. Pruning preserves this state invariant during the invocation; it does not itself change span delivery. The two-view sibling regression asserts both transitions before finalization can recover a sibling from pending or clear the index.

The root cause is that natural End hooks cannot observe children skipped by replay, active-span registries discard completed parent identities, and operation-level deduplication alone cannot distinguish reopened context segments. This is an internal bug fix using existing snapshots and hooks; it adds no public API or configuration. Core scheduling, status mapping, timestamps, provider flushing, testing-library behavior, and conformance handlers are unchanged. Export recovery and broader finalization failure isolation remain #855.

Validation

Implementation validation (Python 3.13.17, Hatch 1.16.5):

• Re-ran the public-handler missing-completion regression against the base OTel source at e42947e8: all 12 cases fail because the callback span is absent.
• hatch run dev-otel:test -q --tb=short: 901 passed.
• hatch run dev-otel:typecheck: 52 source files clean.
• hatch fmt --check in the OTel package and git diff --check: passed.
• Built core and OTel distributions with hatch build.
• hatch run test-wheel-otel:test -q --tb=short: 901 OTel tests and 12 lifecycle cases passed, with installed wheel hashes and source bytes verified.

Coverage includes input/checkpoint/natural deduplication, terminal statuses, concurrent End delivery, unavailable ancestors, actual replay-parent preference, late updates, context-segment re-entry, cached-parent body skipping, unknown updated IDs, warm reuse, overlapping invocations, and child-processor isolation. Added a two-view regression that completing one child preserves its pending sibling, and strengthened the processor-failure test to assert eventual delivery exactly once within the tested invocation.

Existing behavioral expectations are preserved. Internal-state assertions in test_completion_snapshots.py, test_context_segment_reentry.py, test_terminal_input_parent_refs.py, and e2e/test_completion_snapshot_int.py now follow CompletionState; the input-parent helper test expects IDs in a set instead of complete records in a list. The pre-fix failing run above records the regression evidence.

Parent-index validation (Python 3.13.16, Hatch 1.18.1):

• hatch run dev-otel:test -q --tb=short: 1,018 passed, including the local e2e suite.
• hatch run dev-otel:typecheck: 54 source files clean.
• hatch fmt --check in the OTel package and git diff --check: passed.
• With empty-parent pruning removed, the original sibling test still passed in both views. The strengthened test fails in both views when pruning is removed, the parent bucket is deleted unconditionally, or the delivered child ID is retained. All temporary mutations were restored before the full suite ran.

test_natural_child_end_preserves_other_pending_children now asserts the remaining sibling payload and index immediately after the first natural End, then ends the second child and asserts both dictionaries are empty. It checks both child spans and their parent before parent/invocation finalization, and retains the final assertions against duplicate delivery. Existing snapshot and public-handler tests continue to cover completions without natural End hooks. This test-only change does not require rebuilding wheels or changing conformance handlers.

Assumptions and limitations

• Operation parents form the durable service's tree; cyclic input is unsupported.
• Only visible operation metadata can be exported. An updated ID without its operation cannot reconstruct a completion. Service-shaped public-handler fixtures are covered; deployed-service qualification remains unverified.
• Unknown parents fall back to Invocation in Invocation view or a derived logical context in Execution view. Available completed-input IDs can supply logical references after actual replay parents have had a chance to register. A reference cannot recreate a missing span.
• Deduplication is per invocation and does not guarantee exactly-once backend delivery across failed invocations or processor errors. The installed-wheel lane retains the existing testing-dependency selection; that boundary remains [CI]: Validate the built testing-library wheel in OTel compatibility lanes #863.

Implementation decisions

• Use one shared, invocation-owned state class and an ordered ID index, with no additional lock or retry scheduler.
• Retain non-recording contexts after parent completion.
• Preserve parent-first finalization and contain ordinary child-export failures during registration.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.